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HARRY TAMVAKIS 

Abstract. We give a direct proof of the equivalence between the Giambelli 
and Fieri type formulas for Hall-Littlewood functions using Young's raising op- 
erators, parallel to joint work with Buch and Kresch for the Schubert classes 
on isotropic Grassmannians. We prove several closely related mirror identi- 
ties enjoyed by the Giambelli polynomials, which lead to new recursions for 
Schubert classes. The raising operator approach is applied to obtain tableau 
formulas for the Hall-Littlewood functions, the theta polynomials of |BKT2| . 
and related Stanley symmetric functions. Finally, we introduce the notion of 
a skew element w of the hyperoctahedral group and identify the set of reduced 
words for w with the set of standard fc-tableaux on a skew shape X/fJ.. 



0. Introduction 

The classical Schubert calculus is concerned with the algebraic structure of the 
cohomology ring of the Grassmannian G(m, N) of m-dimensional subspaces of com- 
plex afhne iV-space. The cohomology has a free Z-basis of Schubert classes a\, 
induced by the natural decomposition of G(rn, N) into a disjoint union of Schubert 
cells. On the other hand, the ring is generated by the Chern classes of the universal 
quotient bundle Q over G{m,N), also known as the special Schubert classes. The 
theorems of Fieri [Pi] and Giambelli [G] are fundamental building blocks of the sub- 
ject: the former is a rule for a product of a general Schubert class ax with a special 
one, while the latter is a formula equating ax with a polynomial in the special 
classes. When one expresses the Chern classes involved in terms of the Chern roots 
of Q, the Schubert classes are replaced by Schur 5-polynomials, thus exhibiting a 
link between the Schubert calculus and the ring of symmetric functions. 

In a series of papers with Buch and Kresch f BKTl|, IBKT2j , we obtained cor- 
responding results for the Grassmannians parametrizing (non-maximal) isotropic 
subspaces of a vector space equipped with a nondegenerate symmetric or skew- 
symmetric bilinear form. Our solution of the Giambelli problem for isotropic Grass- 
mannians uses the raising operators of Young in an essential way; the resulting 
formula interpolates naturally between a Jacobi-Trudi determinant and a Schur 
Ffaffian. A rather different context in which a Giambelli type formula appears that 
has this interpolation property is the theory of Hall-Littlewood symmetric functions 
[LH IM] ; these objects also satisfy a Fieri rule Mo . 

The raising operator approach allows one to see directly that the Fieri and Gi- 
ambelli results are formally equivalent to each other in all the above instances. This 
amounts to showing that the Giambelli polynomials satisfy the Fieri rule, for the 
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converse implication then follows easily. As a consequence, working either in the 
context of Schubert calculus or the theory of symmetric functions, it suffices to 
prove either of the two theorems to establish them both. In geometry, the Fieri 
rule may be proved concisely by studying a triple intersection of the appropriate 
Schubert cells, following Hodge [H]; this is the method used in }BKTlj . 

When working algebraically, it is convenient to use the Giambelli formula as the 
starting point, and this is the point of view of the present article. One advantage 
of the raising operator definition is that it makes clear that the basic Giambelli 
polynomials, which are indexed by partitions A, make sense when the index is any 
finite sequence of integers, positive or negative. When the index is a partition, it is 
the axis of reflection for a pair of mirror identities^ both closely connected to the 
Fieri rule. The simplest example is in the case of Schur S'-functions: 

SX+a = ^ and ^ SA-a = ^ 

Q>0 fiDX a>Q ti(ZX 

where the sums are over compositions a of length at most ^(A) + 1 and partitions fi 
obtained from A by adding (respectively subtracting) a horizontal strip. In ii4.4l we 
use raising operators and the mirror identities to obtain top row recursion formulas 
for Schubert classes on isotropic Grassmannians. 

Our central thesis up to this point is that the aforementioned results may be 
proved without recourse to their realizations in terms of symmetric functions. The 
main application of the mirror identities, however, lies in the latter theory, where 
they can be used to obtain reduction formulas for the number of variables x which 
appear in the argument of a symmetric polynomial. These in turn lead directly 
to tableau formulas for the polynomials in question. The tableau formulas suggest 
the introduction of new symmetric functions indexed by skew Young diagrams, and 
one can then study their properties. In this way, many important aspects of the 
classical theory of symmetric functions, as presented in the first three chapters of 
[M] . may be established by entirely different methods. Moreover, the intention is 
to apply these techniques in a hitherto unexplored situation. 

Our main result is a tableau formula for the theta polynomials of IBKT2j , which 
are the Billey-Haiman Schubert polynomials |BH] for Grassmannian elements of 
the hyperoctahedral group i?oo (the union of the Weyl groups for the root systems 
of type B„ or C„). The theorem states that for any fc-strict partition A, we have 

(1) eA(x;2/) = 5]2"(^)(x2/)^ 

u 

where the sum is over all fc-bitableaux U of shape A (see H5.3\ for the precise defini- 
tions). It bears emphasis that our proof of ([1]) does not use inner products, tableau 
correspondences, or algorithms such as jeu de taquin and the like. We show that 
formula ([T]) (and its counterpart in the theory of Schur and Hall-Littlewood func- 
tions) follows essentially from the definition of Q\{x;y) using raising operators, 
once the Fieri rule is established. The result is surprising because the Fieri rule for 
the product 'dp ■ Q\ involves partitions fi which do not contain the diagram of A. 

We are led to introduce symmetric functions F^x/\ (^) ' defined for any pair fi C X 
of fc-strict partitions by the equation 

T 
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where the sum is over ah fc-tableaux T of skew shape A//i. When k — 0, the F^^^ 
coincide with the usual skew Schur Q-functions Q\/^. For general fc, we prove that 
^A/)i either vanishes or is equal to a certain type C Stanley symmetric function 
F^^ and therefore is always a nonnegative integer linear combination of Schur Q- 
functions. The function F^ is indexed by an element w of the hyperoctahedral 
group which we call skew. We argue that the skew elements of -Boo are directly 
analogous to the 321-avoiding permutations [BJS| in the symmetric group. For 
example, we show that the reduced words for w are in 1-1 correspondence with the 
standard /c-tableaux of shape A//x. 

We begin this article in 21 by giving a short proof using raising operators of 
the equivalence between the classical Giambelli formula and the Fieri rule. This is 
followed by a discussion of the analogous result for the algebra of Hall-Littlewood 
functions in S}2l and then the more sophisticated arguments of [BKT2j in ij3l The 
raising operators which appear in the Giambelli formulas of isotropic Schubert 
calculus depend on the partition A; this leads to a more challenging and dynamic 
theory. In 21 obtain the mirror identities for Hall-Littlewood functions and 
isotropic Grassmannian Schubert classes, and give some first applications. Section 
[5] contains our treatment of the various tableau formulas. In i)5.1l we stop short 
of proving the (known) tableau formula for skew Hall-Littlewood functions, since 
the more difficult case of theta polynomials and the skew functions is 
handled in detail in %.2\ - i jS.SI Finally, in ij6] we define the skew elements of the 
hyperoctahedral group and relate the results of fj5] to the Billey-Haiman Schubert 
polynomials and type C Stanley symmetric functions. 

For simplicity we will restrict to the type A and C theories throughout the arti- 
cle, and not discuss the analogous results for orthogonal Grassmannians and Weyl 
groups here. We hope that fJTJand the examples of i i3. 21 provide a useful introduc- 
tion to [BKT2j . To emphasize that the formulas in SJT] - SI do not depend on their 
specific realizations in the Schubert calculus or the theory of symmetric functions, 
we have used the nonstandard notation U\, V\, W\ for the basic polynomials, and 
in i il.4[ ij2.31 and ij3. 31 briefly discuss how these relate to other known objects. 

Many thanks are due to my collaborators Anders Buch and Andrew Kresch for 
their inspired work on related projects. 



1. The TYPE A theory 

1.1. Raising operators. An integer sequence or integer vector is a sequence of 
integers {ai}i>o, only flnitely many of which are non-zero. The largest integer 
> such that ^ is called the length of a, denoted l{a); we will identify an 
integer sequence of length i with the vector consisting of its first £ terms. We let 
|a| =^ai and #a equal the number of non-zero parts Ui of a. We write a > (3 
if ai > (3i for each i. An integer vector a is a composition if > for all i and a 
partition if tti > a^+i > for all i. As is customary, we represent a partition A by 
its Young diagram of boxes, which has Ai boxes in row i. We write fj, G X instead 
of /I < A for the containment relation between two Young diagrams; in this case 
the set-theoretic difference X \ fi is called a skew diagram and denoted X/ fi. 

For two integer sequences a, (3 such that \a\ = we say that a dominates (3 
and write a >r /3 if ai + ■ • • + ai > I3i + . . .Pi for each i. Given any integer sequence 
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a = (ai, q;2, ■ • ■) and i < j, we define 

^ [ai, . . . ,at + 1, . . . ,aj - 1, . . .); 

a raising operator R is any monomial in these i?y's. Note that we have Ra a 
for any integer sequence a; conversely, ii a ^ (3, there exists a raising operator R 
such that a = Rf3. See [Ml I-l] for more information. 

1.2. The Giambelli formula. Consider the polynomial ring A = Z[ui,U2,...] 
where the Ui are countably infinite commuting independent variables. We regard 
A as a graded ring with each Ui having graded degree i, and adopt the convention 
here and throughout the paper that uq = I while Ur = for r < 0. For each integer 
vector a, set Ua = Yii '^o-t ; then A has a free Z-basis consisting of the monomials 
u\ for all partitions A. 

Given a raising operator i?, define Rua = una- Here we view the raising operator 
R as acting on the index a, and not on the monomial Ua itself. Thus, if the 
components of a are a permutation of the components of (3, it may happen that 
Rua ^ Rup even though Ua = up as elements of A. Notice that if < for 
£ — ^(a), then Rua — Q va. A for any raising operator R. 

For any integer vector a, define Ua by the Giambelli formula 

(2) Ua■.= Y[il-R^J)ua. 

i<j 

Although the product in ([2]) is infinite, if we expand it into a series of terms we see 
that only finitely many of the summands are non-zero; hence, Ua is well defined. 
For any partition A, we clearly have 

Ux = "A + ^ OAm^m 
M^A 

where ax^ G Z and the sum is over partitions /i which strictly dominate A. We 
deduce that the Ux for A a partition form another Z-basis of A. 

We have U(^r,s) = (1 ^ ^12) u^^.s) — UrUs — Ur+iUs-i, hence ?7(r,s) = — ?7(s_i^r-i-i) 
for any r, s G Z. This property has a straightforward generalization. 

Lemma 1. Let a and (3 be integer vectors. Then for any r^s ^ 1j we have 

U{a,r,s,l3) — —U(^a,s-l,r+l,0) ■ 

We postpone the proof until |j2l where we derive a more general result in the context 
of the Hall-Littlcwood theory (Lemma [3]). 

1.3. The Pieri rule. For any d > 1 define the raising operator R'^ by 

R'^ l[ (l-i?.,). 

l<i<j<d 

For p > and any partition A of length i, we compute 

e 

Up-Ux^Up- R'^ Ux = • J|(l - U(A,p) 

1=1 

= R'+^ ■ + R,j+i + Rl,+, + ■■■) 

i=l 
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and therefore 

(3) Up-Ux^Yl 

where M = J\f{X,p) is the set of all compositions > A such that = \X\ +p and 
i^j = for j > £+1. 

Call a composition v G J\f bad if there exists a j > 1 such that i^j > Xj-i, and let 
X be the set of all bad compositions. Define an involution l : X ^ X a.s follows: 
for v G X, choose j maximal such that Vj > Aj_i, and set 

Lemma [T] implies that Ui, + t/,,(,y) = for every v G X, therefore all bad indices may 
be omitted from the sum in fS]). We are left with the Fieri rule 

(4) Up-Ux^J2u, 

V 

summed over all partitions v with \v\ — |A| +p such that i^i > Ai > > A2 > ■ • • . 
In the language of Young diagrams, this condition states that v ^ \ and the skew 
diagram i//A is a horizontal p-strip. 

Conversely, suppose that we are given a family T\ of elements of A, one for each 
partition A, such that Tq — 1 and the Tx satisfy the Fieri rule Up ■ T\ ~ ^^T^, 
with the sum over 1/ as in (|4]). We claim then that 

Tx^Ux=Y[{l-R^J)ux. 

i<j 

To see this, note that the Fieri rule implies that 

U\ + y^ a\fi = uxi - ■ ■ uxi = 7a + ^ ax^J. Ti_i 

for some constants ax^_i G 1. The claim now follows by induction on A. We thus see 
that the Giamhelli formula and Fieri rule are formally equivalent to each other. 

1.4. The Grassmannian and symmetric functions. Equation Q may be writ- 
ten in the more familiar form 

(5) Ua ^ det{Uai+j-i)^,j■ 
We will prove that ([2]) and ([5]) are equivalent, following ^ L3]. Suppose the length 
of a is £ and let p = {0,1, . . . , £ — 1). We work in the ring Z[xi,Xi^, . . . ,xe, x^^] 
and let Rx°' = x^°' for any raising operator R. The key is to use the Vandermonde 
identity 

det{x{''^)i<ij<i = {xj - Xi) 

l<i<j<i 

which implies that 

n {l~R^J)x"^ n =x"-''det(xr')i<,j<f 

= J2 {~irx",'+^^'^-' ■ ■ ■ x't^^"^'^-' = det(xr-+^-*)i<,,,<,. 

One now applies the Z-module homomorphism 'L[xi,Xi^ , . . . ,xi,x'^^\ — > A sending 
to Ua for each a to both ends of the above equation to obtain the result. 
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The two standard realizations of the Fieri and GiambeUi formulas in the liter- 
ature are the cohomology ring of the Grassmannian G — G(m, N) and the ring A 
of symmetric functions. The ring H*(G,Z) has a free Z-basis of Schubert classes 
a\, one for each partition A whose diagram is contained in an m x (N — m) rec- 
tangle TZ{m,N — m). There is a ring epimorphism (p : A H*(G,Z) sending the 
generators Up to the special Schubert classes ap for 1 < p < N — m and to zero for 
p > N — m. The map satisfies <t>{U\) = a\ M \ G TZ{m, N — m), and <j){U\) = 
otherwise. For further details, the reader may consult e.g. |Fu[ Chapter 9.4]. 

Let X = (xi,X2,...) be an infinite set of variables, and for each p > let 
hp — hp{x) be the p-th complete symmetric function, that is, formal sum of all 
monomials in the Xi of degree p. The ring A = Z[ft,i, h2, . . .] may be identified with 
the ring of symmetric functions in the variables Xi. There is a ring isomorphism 
A A sending Up to hp for all p. For any partition A, the element Ux is mapped 
to the Schur S*- function s\{x). The equation 

s\{x) = det(/iAi+j-j(a;))i<ij<£(A) 

expressing the Schur functions in terms of the complete symmetric functions is 
called the Jacobi-Trudi identity (see e.g. |Mi I- (3.4)]). 



2. The Hall-Littlewood theory 

2.1. The Giambelli formula. Let vi,V2, ■ ■ ■ be an infinite family of commuting 
variables, with Vi having degree i for all i. As before let vo ~ 1, = for 
r < 0, Va = Yii'^ai for each integer sequence a, and Rva = VRa for any raising 
operator R. Let t be a formal variable and consider the graded polynomial ring 
At=Z[t][vi,v2,...]. 

For any integer vector a, define Va G At by the Giambelli formula 



(6) ^"^^Hr 



Rij 



— tRij 

Let a — (ai, . . . , ag-i) be an arbitrary integer vector and r G Z. Expanding the 
raising operator product in ^ 

i—i 

nl — Rij _ TT 1 — Rij TT 1 — Rii 
1 - ~ T^^tR~ ^^T^~tR~i 

i<i<j<e ■' i<i<j<e ■' 1=1 

along the last (i.e., the £-th) component of (a,r) gives 

(7) V^a,r) = E - l)#^K.+^«.-|^|, 

7 

summed over all compositions 7 G N^~^, where N ~ {0, 1, . . .}. Equation ([7]) may 
be used to give a recursive definition of Va ■ The v\ and V\ for A a partition form 
two free Z[t] -bases of Af. 

Proposition 1. Suppose that we have an equation in At 
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where the sums are over all v = (y\^ . . . , vi), while and b^, are polynomials in 
Z[t], only finitely many of which are non-zero. Then we have 



for any integer vectors fi. 



Proof. It suffices to show that if c^V^ = for some c„ G Z[t] , then J2i, V(^.^) = 
0. We will prove that c^V(p_y) = for any integer p; the desired result then 
follows by induction. 

Upon expanding the raising operators in the definition of V^, the equation 

X]i/CiyV,y = becomes J^v^'^''^'^ ~ ^ ^'^^ some e We therefore must show 

^+1 ^ _ ^ 

that '^^Ci/D v/p = 0, where -D = I I —. By exchanging rows, we may 

assume that both ^^'cvV^, and CV-D W(p^) are summed over decreasing integer 
sequences v = (y\^ . . . ,vi)^ because 

^ '^{P-y) = X! - l)*^«p+|7|«i'-7 = ^ "(p,t(i^)) 

7>0 

for every permutation t in the symmetric group Sg. \iv has a negative component 
then clearly Vi, = Dv(ji ,j) = 0. Moreover, since the v^^ for v a partition form a 
Z[t] basis of At, it follows from '}2i^c^Vi, = that c^ — Q for all partitions v. We 
therefore have Ci/D f (p,jy) = 0, as desired. □ 

For any integers r and s, we claim that the equation 

(8) V(r^s) + = i (^(r+l,s-l) + ^(s,r)) 

holds in the ring At. Indeed, ([5]) follows from the identity 

= {l-Ri2) + t- — -^Rl2 



1 — tRi2 1 — tRi2 

and the fact that (1 — Ri2){v[r,s) + i^(s-i,r+i)) = in At. 
Lemma 2. For c G Z and d > I, we have 

d-l 

(9) Vic,c+d) + (1 ~ X/ ^(c+ix+d-i) = * V{c+d,c) ■ 

2=1 

Proof. We use induction on d. When d = 1 the result follows by setting s = r + 1 
in ([8]). If d > 1, the inductive hypothesis gives 

(1 — t)''^V{c+i,c+d~i) = V(c+d-l,c+l) ~ tV(c+l,c+d~l)- 
1=1 

The identity ([9]) thus reduces to 

y{c,c+d) + V(c+d-l,c+l) ~ tV(c+l.c+d-l) = tV(c+d.c), 

and this follows directly from equation □ 
We can generalize the identity ^ as follows. 
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Lemma 3. Let a and [3 be integer vectors. Then for any r, s £ Z, we have 

(10) y{a,r,s,fj) + V(a,s-l,r+l,f3) — t {V(^a,r+l,s-l,[3) + V(a,s,r,/3)) 

in the ring At . 

Proof. By Proposition[l] we may assume that a is empty. If /3 — (/?', b) has positive 
length, where & £ Z, we set ^ = (r, s, /3') and the identity follows by induction, since 

7 

Finally, if both a and /3 are empty, the result is true by ([8]) . □ 

Notice that Lemma [1] is the specialization of Lemma [3] at t = 0. Using Lemma [3] 
and the argument in the proof of Lemma [2] gives the following result. 

Corollary 1. For c G Z, d > 1, and a, (3 any integer vectors, we have 

d-l 
1=1 

2.2. The Pieri rule. Let A C /i be two partitions such that /i/A is a horizontal 
strip, and let J be the set of integers c > 1 such that ^/A does not (resp. does) 
have a box in column c (resp. column c + 1). Define 

ce.7 

where TOc(A) denotes the number of parts of A that are equal to c. 
For any d > 1 define the raising operator i?^ by 

11 1^7]^- 

l<i<j<d 

Given a partition A of length £, we compute 

1 — tRi^i+ 



Vp-Vx^Vp- vx = Rt ■ I I -j 5 W(A.p) 

= ■ + (1 - i)i?,,,+i + (1 - t)i?2,^i + • • ■ ) 

and therefore 

(11) Vp-V>.^Y.^\- i)#(''-^) K, 

where N — Af{X,p) is the set of compositions > A such that = |A| +p and 
i^j = for j > £ + 1, as in gLl 

For any integer composition ly, set i^* — (1^2, 1^3, ■ • ■)• Let TV* = {i^ G A/" | i^j < 
Aj_i for j > 2} and for G A/"*, define 

r, = (i-t)#(^i-^i)V.,./A-WK. 

Using Proposition [1] and induction on the length of A gives 

(12) E (1 - t)*'^''-^^ K - 51 
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Note that if e Af* satisfies 1^2 < ^i, then Ti, ~ ipu/x{t) K/- Therefore 

(13) E E ^^/aWk+ E 



We claim that for each fixed d > 0, 

(14) E = E ^-/a(ok. 



Equation p4)) is justified by using Corollary [T] with a = to fix all rows except the 
first two. We then apply the identity 

d-l 

(15) y(c,c+d) + (1 ^ ^) E ^ic+hc+d-i) + (1 - i) V(c+d.c) = V{c+d,c) 

i=l 

when \ A has a box in column c = Ai, and the identity 
(16) 

(1-i") V(,,e+d) + (l-t)(l-i™) J2 V(c+,,c+d-r) + {l-t) T/(c+dx) - (1-^™+') V^,+d,c) 

i=l 

for m > 1, otherwise. The identities ([T5)l and (|16p follow directly from Lemma [21 
proving the claim. 

Summing ([T^ over all d > 0, applying the result to p^ . and taking (jlip and 
P2p into account, we finally obtain the Fieri rule 



(17) Vp-Vx=Y.i^^/xit)V^, 

summed over all partitions /i D A with = |A| +p and fi/X a horizontal p-strip. 

Conversely, suppose that we are given a family T\ of elements of At, one for 
each partition A, such that Tq = 1 and the Tx satisfy the Fieri rule Vp ■ T\ — 
^ '0^/;)j(i)r^, with the sum over and i^f^/xit) as in (I17p . It follows then as in 
go] that 

We have established that the Giambelli formula ^ and Fieri rule (fTT)) are formally 
equivalent to each other. 

2.3. Hall-Littlewood symmetric functions. Let a; ~ {xi, X2t ■ ■) be as in tjl.41 

define the formal power series qr{x] t) by the generating equation 

1 — Xjtz 



1 — XiZ 



and set Ff = Z[i][(7i, (72, • ■ ■]• There is a Z[t]-linear ring isomorphism At —>■ Ft 
sending Vr to qr{x;t) for all r. For any partition A, the element Vx is mapped to 
the Hall-Littlewood function Qx{x]t). The raising operator formula ([6|) for Hall- 
Littlewood functions is due to Littlewood ^i] , who also obtained Lemma |3] in this 
setting. The Fieri rule lfT7|) for the functions Qx{x]t) was first proved by Morris 
|Mo| : an alternative proof may be found in [M, IIL5]. 
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If A denotes the ring of symmetric functions from §1.41 then the Z[t]-hnear ring 
isomorphism At — > A[i] sending Vr to hr(x) for every r > 1 maps Vx to the modified 
Hall- Littlewood function Q\{x; t) of [LLTj . Now speciahze t = —1 in the definition of 
the ring At and its distinguished basis V\, and consider the isomorphism — > A 
obtained by mapping Vr to the r-th elementary symmetric function er{x) for every 
r > 1. The basis element V\ is then mapped to the Q-polynomial Qx{x) of Pragacz 
and Ratajski pRj . 

3. The type C theory: Isotropic Grassmannians 

Fix an integer fc > 0. The aim of this section is to explain the main conclusions of 
[BKTll IBKT2| regarding the Giambelli and Fieri formulas for the Grassmannian 
IG = IG(n — k,2n) of isotropic {n — /c)-dimensional subspaces of C^", equipped 
with a symplectic form. We provide an exposition of those aspects relevant to the 
present article, including examples, and refer the reader to the original papers for 
detailed proofs. 

3.1. The Giambelli formula. We consider an infinite family wi, W2, ... of com- 
muting variables, with Wi of degree i for all i, and set w^j — 1, Wr ^ Q for r < 0, 
and Wa — Yli as before. Let I^''^ C W2, . . .] be the ideal generated by the 

relations 

(18) ^W,rr) = +2y^{-lYWr+iWr-i = for T > fc. 

1 + -Kl2 ^ 
i—l 

Define the graded ring B'-'^^ — Z,[vi,V2, ■ ■ .]//*^'^^. All equations in B^''\ such as the 
Giambelli and Fieri formulas, will be valid only up to the elements of I'^'^K 

Let A° = G N X N I 1 < I < j}, equipped with the partial order < defined 

by {i' if) 1^ {hi) if E^'^d o^^ly if < i and f < j. A finite subset D of A° is a 
valid set of pairs if it is an order ideal, i.e., (i,j) G D implies {i',j') G D for all 
(i',j')GA° with (z',/) < (*,j). 

A partition A is k-strict if all its parts greater than k are distinct. Given a 
/c-strict partition A, we define a set of pairs C(A) by 

C(A) = {{i,j) G A° I Ai + Aj > 2fc + j - i and j < £(A)}. 

It is easy to check that C(A) is a valid set of pairs. Conversely, assuming A: > 0, let 
D be any valid set of pairs D and set di = #{j | («, j) G -D}. Then the prescription 

^ _ ffc + 1 + d, if > 0, 
' \k ifd, = 

for 1 < z < di + 1 defines a fc-strict partition A such that C(A) = D. The white 
dots in Figure [T] illustrate a typical valid set of pairs. 

For any valid set of pairs D, we define the raising operator 

If A is a /c-strict partition, then set R^ := R'^^-^'> . For any such A, define W\ G B^'''> 
by the Giambelli formula 

(19) Wx-.^R^wx. 
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1 — Rij 

Wx. 



R. 
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Figure 1. A set of pairs (in white) and its outside rim (in grey). 

In contrast to ([2]) and ([6]), we see that the raising operator i?^ in the Giambelh 
definition (fT9|) depends on A. When Ai + Aj < 2k + j — i for all i < j, equation (fT9|) 
becomes W\ — Yli^j{^—Rij) wx, while when Xi+Xj > 2k+j—i for 1 < z < j < ^(A), 

the formula is equivalent to Wx = Y[ J 

i<j 

Example 1. In the ring we have 

^621 = , p . , p (1 - ^23)^621 

= (1 - 2i?i2 + 2Rl^ )(1 - 2i?i3 + 2i?23 )(1 - i?23) U;621 

= (1 - 2i?i2 + 2Rl^ - 2i??2)(l - 2i?i3 - R23) 

= W621 - WQ3 - 2^711 + 4W81 - 2wq 

= W()W2Wi — wqw^ — 2w7wl + AwsWi — 2109. 

From equation we deduce that either the partition A is fc-strict, or wx is a 
Z-linear combination of the such that /j, is fc-strict and jj, )^ X. It follows that the 
Wx for A fc-strict span as an abelian group. A dimension counting argument 
shows that in fact, the wx for A fc-strict form a Z-basis of B^''\ From the definition 
^ it follows that for any partition A, Wx is of the form 

Wx ^ Wx + ^ flA^ 



with coefficients ax^i G Z. We deduce that 

Wx ^ wx + ^ bx^, Wfj, 

with the sum restricted to fc-strict partitions fi >- X. Therefore, the Wx as A runs 
over fc-strict partitions form a Z-basis of B^''\ 

More generally, given a valid set of pairs D and an integer sequence a, we denote 
R^ Wa by W^ . We say that D is the denominator set or D-set of W^ . If r, s G Z, 
then 

^^{r,s) - '''^(s-l,r-|-l) 



while if 7^ and r + s > 2fc, then 



WP = -WP ^ 
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Figure 2. Two fc-related boxes in a fc-strict Young diagram 

in the ring B^'^'. To generalize these equations, care is required, as e.g. the direct 
analogue of Lemma[T]for the fails. Suppose D is a valid set of pairs and j > 1. 
We say that the pair + 1) is D-tame if 

(i) (jJ + 1) ^ and for all h < j, {h,j) ^ D if and only if {h,j + 1) ^ D, or 

(ii) Ui j + 1) G -D and for all h > j + I, {j, h) e D if and only if (j + 1, /i) G D. 

Lemma 4 f [BKT2j ). Let a = (ai, . . . , a^-i) and (3 — {l3j+2, ■ ■ ■ , Pi) be integer 
vectors, and assume that + 1) is D-tame. 

(a) // + 1) ^ D, then for any r, s £ Z we have 

(b) // {j, j + 1) G D, then for any r, s £ Z, such that r + s > 2k, we have 

^(Q,r,s,/3) = ~^(Q,s,r,/3) ■ 

Example 2. The ring B := B^*'^ has free Z-basis consisting of the Wx for A a strict 
partition, and the Giambelli formula (|19p is equivalent to the identity 



(20) W, = l['^^w,. 



■ ^3 

Lemma U in this case gives W(a.,r,s.fj) = ~W^(Q.s.r,/3) whenever r + s > 0. 

3.2. The Pieri rule. We begin our analysis in the same way as in the previous 
sections. For p > and any fc-strict partition A of length I, we find that 

(21) ^t^C(A)^ 

where M = Af{X,p) is the set of all compositions ly > X such that |z/| = |A| -l-p and 
^^/ = for j > £+1. The problem with the right hand side of pT|) is two- fold: first, 
the compositions i> G Af are not fc-strict partitions, and second, the denominator 
set C(A) is the same for each term in the sum, and will have to be modified so as to 
agree with the summands in the eventual Pieri rule. Rather than give the complete 
argument, we will state the Pieri rule which results and then discuss some features 
of the proof. 

We let [r, c] denote the box in row r and column c of a Young diagram. Suppose 
that c < k < c' . We say that the boxes [r, c] and [r',c'] are k-related if c -1- c' — 
2k + 2 + r — r' . In the diagram of Figure [H the two grey boxes are fc-related. 

Given two partitions A and /x with A C /u, the skew Young diagram /z/A is called 
a vertical strip if it does not contain two boxes in the same row. For any two fc-strict 
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partitions A and /i, let ai (resp. (3i) denote the number of boxes of A (resp. fi) in 
column i, for 1 < i < k. We have a relation A ^ /i if can be obtained by removing 
a vertical strip from the first k columns of A and adding a horizontal strip to the 
result, so that for each i with 1 < z < fc, 

(1) if Pi = ai, then the box [ai,i\ is fc-related to at most one box of /i \ A; and 

(2) if (3i < ai, then the boxes [(3i,i],. . [ai,i\ must each be /c-related to exactly 
one box of /i \ A, and these boxes of /i \ A must all lie in the same row. 

li X ^ fi, we let A be the set of boxes of \ A in columns k + 1 and higher 
which are not mentioned in (1) or (2). Define the connected components of A by 
agreeing that two boxes in A are connected if they share at least a vertex. Then 
define A^(A, n) to be the number of connected components of A which do not have 
a box in column k + 1. Finally, we can state the Fieri rule for B^^): Given any 
/c-strict partition A and integer p >Q, 

(22) Wp-Wx^ 2^^^''^)!^^. 

A^/i, |/i| = |A|+p 

The outside rim dC oi C = C(A) is the set of pairs € A° \ C such that i = 1 
or (i — 1, j — 1) G C; an example of these sets is displayed in Figure [1] The Young 
diagrams of the partitions /i which appear in ((22l) do not all contain the diagram of 
A, in contrast to the Fieri rule of §2.21 for Hall-Littlewood functions. However, the 
denominator sets C(^) for the partitions fi with A — > /.t all contain C, are contained 
in C U 9C, and are empty beyond column £+ 1. 

To prove (|22p . our task is to show that 

(23) W^^^'^^ E 2^(^-'^) ly^c(^) . 
vej\f(x,p) x^fj., \fj.\=\x\+p 

It is clear that we need a mechanism to modify the denominator sets of the terms 
Wu on the left hand side of ([^5]) . This is based on the observation that if Z) is a 
denominator set and («, j) € A° \ D, then 

(24) = M/f -(-^■) + <^1^-) . 
Equation (|24p follows directly from the raising operator identity 

1 + Rij 1 + Rij 

The following three detailed examples illustrate how Lemma |4] and ((24|) may be 
used repeatedly to obtain ([23]) . For simplicity, the commas are omitted from the 
notation for integer vectors and pairs. 

Example 3. The following chain of equalities holds in B^^\ 

— _L ( Wl2,13 , wl2,13\ ( 11712,13 , TT.12,13A 

— 1^^^32111 + \^*^'^3221 + *^'^4211 J + \^*^>^4211 + *^>^5201 J 

— Wl2 I ('Wl2. 13,23 , TX7l2,13,23N , g wl2,13 , /t^12,13,14 t^12,13,14x 

— '''^32111 + 1*^^^3221 + "^^3311 ) + ^ '^'^4211 + l'^'^5201 + '^'^6200 ) 

_ ^C(32111) 2 j4^C(4211) ^C(62) 

— '^'^32111 ^ '^'^4211 ^ '^62 

Observe that LemmaSl^a) is used to show that W^2i2 ~ ^5^2^61'^'^"' = O7 while Lemma 
Hb) implies that W^^ = M^322l'''' = W^33n'''' = «• 
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Example 4. The following chain of equalities holds in B^^\ 
+ W^®2i + Wl^ + l^lai + iy|22 + W^li3 

^12\ I /'ti/12 I TJ/12 \ I /tt/12 



+ (W^24 + Wll) + (T^gili + Wll,) + (M^^^li + Wli,) 



^12,13^ 



Wli, + (W^in" + W^5i ' 



Here we use Lemma |4l[a) to see that 

<2 = W^l22 + <3 = W^5o¥' = 0, 

while Lemma |3^b) gives 

Wli = wli + w\l = Ty2'li + wli, = 0. 

Example 5. Let fc = 0. We give a complete cancellation scheme along the above 
lines assuming that the integer p is less than or equal to . For any integers d, e > 1 
define the raising operator 

We compute that 

ueJ\f veM j=i 

and therefore 

(25) Wp-Wx= ^-+7 

(i^,7)6A/''(A,P) 

where Af'{X,p) denotes the set of all pairs {v, 7) of integer vectors of length at most 
£+1 with ly e Af, V + j > X, \iy + j\ ^ \X\ + p, and 7, G {0, 1} for 1 < i < £. 

For every pair (i/, 7) e M'{X,p), define /.t = + 7. Call a pair {v, 7) feaii if there 
exists a j > 1 such that 

(i) ^ij — Hj-i or (ii) /ij > Aj_i or (iii) — Aj_i and 7j = 0. 
Let X be the set of all bad pairs in M' {X, p) . We next define an involution l : X X 
as follows. For (2^,7) £ X, choose j maximal such that (i), (ii), or (iii) holds. If 
= ^J,j-l, we let (i^',7') = (2^,7). Otherwise, let zu £ Si he the transposition 
(j-l,j) and define ((^',7') by setting (j^-,7-) = (j^^j,(j), 7^(1)) for 1 < i < £. Finally, 
set 7) = {v' , 7') for each {ly, 7) G X. Lemma 2] applied to rows j — 1 and j gives 
VFi,+-y = ~Wyi+^i for every (i^, 7) G X, therefore all bad indices may be omitted 
from the sum in (|25p. 

We are left with pairs {v, 7) such that /i = i^ + 7DAisa strict partition with 
/i/A a horizontal strip. Observe that every connected component C oi fx/X which 
does not lie in row £ + 1 contributes a multiplicity of 2 to the sum (^5]) . Indeed, if 
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C lies in rows r through s with r < s, then condition (iii) imphes that 7^ = 1 for 
r < i < s, while 7r can be either or 1. We have therefore proved the Fieri rule 



summed over all strict partitions /i D A with = |A| +p and /i/A a horizontal 
p-strip. The exponent A^(A, /i) equals the number of connected components of ///A 
which do not meet the first column. 

In the general case, there is a substitution algorithm by which the left hand side 
of (|23p evolves into the right hand side. Although Examples [21 HI and [5] appear 
encouraging, they are still far from a precise formulation of this algorithm for a 
general Fieri product Wp ■ W\. It is instructive try to prove along these lines that 
the Fieri rule of Example [5] holds without the simplifying assumption that p < Xi. 
The complete proof for arbitrary fc > is given in 'BKT2]; we note here that for 
technical reasons, the argument is performed in type B, that is, for odd orthogonal 
Grassmannians. Each summand Wi, for v £ M \n (|23p gives rise to a tree of 
successor terms, with the branching given by repeated substitutions using (|24p . In 
this way, we obtain the substitution forest] the roots of the trees in the forest are 
the Wv for v G M. By construction, the sum of all the roots is equal to the 
sum of all the leaves. When the algorithm terminates, the set of leaves contains 
the terms W/f for A ^ /i, each appearing exactly 2^''*''^^ times; the remaining 
leaves are either zero or cancel in pairs, as dictated by Lemma 21 

Once we know that the Giambelli formula pj]) used to define W\ satisfies the 
Fieri rule ([22]), the same reasoning as in the previous two sections establishes that 
the two results are formally equivalent. 

3.3. The isotropic Grassmannian and theta polynomials. Let the vector 
space E = C^" be equipped with a nondegenerate skew-symmetric bilinear form. 
A subspace S of E' is isotropic if the form vanishes when restricted to S. The 
dimensions of such isotropic subspaces S range from to n; when dim(E) = n we 
say that S is a Lagrangian subspace. 

Choose n > fc > and let IG = IG(n — fc, 2n) denote the Grassmannian 
parametrizing isotropic subspaces of E. Its cohomology ring II*(IG,Z) has a free 
Z-basis of Schubert classes cta, one for each fc-strict partition A whose diagram is 
contained in the (n — fc) x (n + fc) rectangle Tl{n — k,n + k). The special Schubert 
classes Up for IG are the Chern classes of the universal quotient bundle over IG, as 
in m.4\ There is a ring epimorphism -0 : B^'^^ — > H*(IG, Z) sending the generators 
Wp to the special Schubert classes cTp for 1 < p < n + k and to zero for p > n + k. 
For any fc-strict partition A, we have ip(W\) = cta if A C TZ{n — k,n + fc), and 
i^iWx) = otherwise. 

Let X = (xi, X2, ■ ■ ■) as in §1.41 and set y = (yi, . . . , yk). Define the formal power 
series ■dr{x,y) by the generating equation 



Set r^'^' = 'L[^l^■^2^ ■ ■ ] to be the ring of theta polynomials. There is a ring iso- 
morphism B^*^) r^'') sending Wp to dp for all p. For any fc-strict partition A, the 




fc 
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element Wx is mapped to the theta polynomial Q\{x ; y) of |BKT2j . These polyno- 
mials agree with the type C Schubert polynomials defined by Billey and Haiman 
[BH] indexed by a Grassmannian permutation of the hyperoctahedral group; see 
[BKT2i §6] and ||6]of the present paper for further information. 

We next discuss this theory when fc = 0. If a has length £ and m > is the least 
integer such that 2m > then equation pop may be written in its original form 

(26) Wa = Pfaffian(W/„, 

,Qj ) l<i<j <2m • 

To see this, one may argue as in ijl.41 this time using Schur's identity [Si §IX] 



TT ^ Pfaffian 

±1. X, J- 'T - 



l<r<s<2m 

In the theory of symmetric functions, the ring r := r(") is called the ring of Schur 
Q-functions. For any strict partition A, the element W\ is mapped to the Schur 
Q-function Q\{x), which Schur "W] defined using the Pfaffian equation (I26p . 

Let (j) denote the ring homomorphism A-i —^ T sending Vp to qp{x) for every 
p > 1. Then 4>{Vx) — Q\{x) whenever A is strict, and (l){V\) ~ for non-strict 
partitions A. The Pieri rule for the products qp ■ Q\ may therefore be obtained by 
specializing Morris' rule p7p for the Hall-Littlewood symmetric functions when t = 
— 1. The Pieri and Giambelli formulas for the Lagrangian Grassmannian LG(n, 2n) 
were proved by Hiller and Boe [HBj and Pragacz [Praj , respectively. 

4. Mirror identities and recursion formulas 

4.1. Let A be any partition of length I. Our proof of the Pieri rule for V\ in ij2.2l 
began with the equation 

Vp-Vx^Vp- Rvx^ Rv(x._p) 
for an appropriate raising operator i?, and relied on the identity 

(27) E(1-*)^"^^+"=E^maW^m 

where the first sum is over all compositions a of length at most ^ + 1, and the 
second over partitions /i D A such that /z/A is a horizonal strip. On the other hand, 
we may write Vp ■ Rvx = R' V(^p^x) for a raising operator R' which agrees with R 
up to a shift of its indices, and attempt a similar analysis. When p is sufficiently 
large, we are led to the mirror identity to (j27p . which is an analogous formula for 

the sum '^{l-t)*°'Vx^a. 

a>a 

Suppose /I is a partition with /i C A such that X/fj, is a horizontal strip. Let / 
denote the set of integers c > 1 such that A//i does (resp. does not) have a box in 
column c (resp. column c+ 1). Define 

^x/,{t) = l[(^-t'"^^'^) 

cei 

and observe that for p > Ai, the Pieri rule (|17p for Vp ■ Vx may be written in the 
form 

(28) Vp-Vx=J2 E ^A/M(0^(p+r,^)- 
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Theorem 1. For any partition \, we have 

(29) ^(l-t)#"yA_„^^^VMW^M, 

Q>0 ^^<z\ 

where the first sum is over all compositions a, and the second over partitions /i C A 
such that X/fi is a horizontal strip. 

Proof. Choose p > \X\ and let £ = Expanding the GiambcUi formula with 

respect to the first row gives 

Vp-Vx = Ri+' n T^^^(p.A)=^r' l[{l + {l-t)R,, + il-t)Rl + ---)v^p^,^ 

J=2 3=2 

and therefore 

(30) Vp-Vx = Y,{l-t)*'-V(p+\^\^x-c.)- 

Q>0 

We compare (|28l) with (f30|) . and claim that for every integer r > 0, 



(31) 5] (1 - = '^A/M(*)n 



I CK I —7^ M C A 

' ' |,<l = |A|-r 



The proof is by induction on r, with the case r = being a tautology. For the 
induction step, suppose that we have for some r > that 



(32) 5^ 5^(1-0*" ^(p+..A-.)-E E '^V.(0^( 



(p+s,/i) • 



- I I - |,.| = |A|-3 



We will require an auxiliary result which stems from |BKT3] . 

Lemma 5. Let Pj. be the set of partitions /i with |^| = r, and let m he a positive 
integer. Then the 7i[t]-linear map 

: Z[t] ^ A, 

which, for given r and fi G P,., sends the corresponding basis element to Wm-r^? 
is injective. 

Proof. The Fieri rule (|17p implies that the image of (/) is contained in the linear 
span of the elements V(m-r.p) foi' < r < ^ and /i in P^- Now the linear map (j) is 
represented by a block triangular matrix with invertible diagonal matrices as the 
blocks along the diagonal, and hence is an isomorphism onto its image. □ 

Expanding the Giambelli formula with respect to the first component, we obtain 
Vp^s V\-a as the leading term of V^^p^g.x-a) i while Vp+s is the leading term of 
^(p+s./j)- Using these in ([5^ together with Lemma [5] proves that 

(33) E (1 - ^)*" = E ^A/^(*)^M- 

I Ct I — C A 
' |,<l = |A|-r 

Proposition [1] now shows that (|3ip is true, completing the induction. Since we 
have simultaneously checked that ([55)1 holds for every integer r > 0, the proof of 
Theorem [T] is complete. □ 



18 



HARRY TAMVAKIS 



4.2. Let A be any fc-strict partition of length I. In this section, we wiU obtain the 
mirror identity to the foUowing version of (|23|) : 

Q>0 A-»/i 

where the first sum is over all compositions a of length at most ^+1, and the second 
over fc-strict partitions jj. with A — > /z. The next result is an easy consequence of 
the Fieri rule ((22l) . 

Lemma 6 f [BKT3j ). Let\ andv be k- strict partitions such that vi > max(Ai,£(A) + 
2k) and p > 0. Then the coefficient of in the Fieri product Wp ■ Wx is equal to 
the coefficient of Wt^y^j^i ,^^ y^ ) in the product Wp+i ■ Wx- 

Lemma |6] is used to make the following definition. 

Definition 1. Let A and ^ be fc-strict partitions with ^ C X, and choose any 
p > max(Ai + 1, ^(A) + 2k). If | A| = |/-t| + r and A ^ {p + r, fi), then we write /i A 
and say that A//z is a k-horizontal strip. We define n{\/ ^) := N(X, (p + r,fi)); 
in other words, the numbers n{\/ ji) are the exponents that appear in the Fieri 
product 

(34) zi;,-VFA = ^2"(V^)lF(p+,,^) 

with the sum over integers r > and fc-strict partitions /i C A with = |A| — r. 

Note that a fc-horizontal strip A//i is a pair of partitions A and /i with /i -w A. 
As such it depends on A and /i and not only on the difference A \ /i. A similar 
remark applies to the integer n(A//i) and the polynomials ipx/i_i{t), ^\/fj,{t) of §4.11 
Observe also that n(A/A) — and n(A//i) > 1 whenever \^ p.. We will study the 
relation /i A and the integer n{X/ p) in more detail in ij5.3l 

1 - i? . 

Proposition 2. Let D = ^ ^ , and suppose that we have an equation 



E 



in B^^\ where the sums are over all v = (i^i, . . . , vg), while and are integers 
only finitely many of which are non-zero. Then we have 



in the ring fiW, for any integer p. 



Proof. It suffices to show that "^^CiyWi, — implies that c,yD W(^p ,^) = 0. We 
may assume by interchanging rows that the sum is over partitions v, because 

Dwi^p^^,) = DW(p^r{i')) 

for every permutation r in the symmetric group Si, and w^, = Dwi^p^,y) = when- 
ever V has a negative component. Recall that the for fc-strict u form a Z-basis for 
B'^'^\ Using the relations (|18p . we see that for any partition v, either v is fc-strict, or 
Wv is a Z-linear combination of the such that /i is fc-strict and ji )^ \. It follows 
that we can systematically identify the sum CuWu with a linear combination of 
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relations, corresponding to successive rows h and h + 1 oi ly with Vh = i^h+i > k. 
Therefore it wiU suffice to show that a relation — ^ Wi, = implies that 

1 + Rh,h+1 

jj h,h+i _ Q Pqj. ]^ jg enough to check that 



f + Rh.h+l 



1 — Rl,h 1 — Rl,h + l 1 — RhM+1 



1 + RlM 1 + RlJi+1 1 + RhM+l 

But we have 

1 — Rl,h 1 — Rl,h+l f — Rh.h+l _ f — Rl.h 1 — -Rl,/i+l _^ f — Rh,h+1 
1 + Rl,h 1 + Rl,h+1 1 + RhM+1 1 + Rl,h 1 + Rl,h+1 1 + Rh,h+1 

and since Vh = f^/i+i, the result is clear. □ 

Theorem 2. For A any k-strict partition we have 

(35) ^f-a H 2"(^/^) W^^, 

where the first sum is over all compositions a. 

Proof. The argument is essentially the same as that in the proof of Theorem [TJ We 
choose p > |A| + 2fc and prove by induction that for each r > 0, 

a|— r ^^-^x 

where C = C{{p, X)). Clearly we have an analogue of Lemma [5] which holds for the 
algebra B^''^ and we use Proposition [2] as a substitute for Proposition [1] □ 

4.3. We now give some consequences of the previous mirror identities for the Schur 
S- and Q-functions. Given any integer sequence u, let s^{x) and Q^{x) denote the 
Schur functions defined in §2.31 and §3.3( respectively. When f = 0, Theorem [1] 
specializes to the following well known result. 

Corollary 2. For any partition \, we have 

E sx-a{x) = E*m(2;), 

Q>0 M 

where the first sum is over all compositions a and the second over all partitions 
/I C A such that X/ fi is a horizontal strip. 

The rim of a partition A is the set of boxes [r, c] of its Young diagram such that 
box [r + 1, c + 1] lies outside the diagram of A. If we choose k > |A| in Theorem^ 
then we have C(A) =0 and can thus deduce the following result. 

Corollary 3. For any partition A, we have 



Q>0 



where the first sum is over all compositions a and the second over all partitions 
H C X such that A/// is contained in the rim of X, and n(A//i) equals the number of 
edge- connected components of X/fj,. 
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The shifted diagram of a strict partition A, denoted 5(A), is obtained from the 
usual Young diagram by shifting the i-th row (i — 1) squares to the right, for every 
i > I. Moreover, when /i is a strict partition with /i C A, we set S{X/fi) = S{X) \ 
iS(/i). We introduce this notation to emphasize the similarity between Corollary [3] 
and the next result, which follows by setting fc = in Theorem [2l 

Corollary 4. For any strict partition X, we have 

a>0 M 

where the first sum is over all compositions a and the second over all strict partitions 
jjL <Z X such that 5(A//i) is contained in the rim of S{X), and n^X/n) equals the 
number of edge-connected components of S{X/ ii). 

4.4. We next show how the raising operator formalism we have developed may be 
used to obtain a recursion formula for the basis elements Wx, expressed in terms 
of the length of A. This question came up naturally during our work on Giambelli 
formulas for the quantum cohomology ring of isotropic Grassmannians |BKT3| §1.3] . 

Throughout this subsection A is a partition of length £ and p > Ai is an integer. 
For completeness, we begin with the kind of recursion we have in mind for the 
elements C^(p,A) of fJTJ We claim that for any partition A and p > Ai, we have 

(36) U^p^x)^Y.^-irup+rU^ 

where the sum is over integers r > and partitions /i obtained from A by removing 
a vertical strip with r boxes. Indeed, Applying the definition ([2]) gives 

U{p,x) = n (1 - ^-^i) 11(1 - i?i,) u^^^^x) = E(-l)' 

j>i>l j>l r,^ 

where the sum is over integers r > and compositions v obtained from A by 
removing r boxes, no two in the same row. If u is not a partition, then we must 
have Vj+i = Vj + 1 for some j; hence U,y ~Qhy Lemma[T] The result follows. 

The analogue of equation (j36|) for the W^(p,A) when p and A are arbitrary appears 
complicated. The next theorem gives a top row recursion for sufficiently large p. 

Theorem 3. For any k-strict partition X and p > max(Ai + 1, ^(A) + 2k), we have 

(37) W^(p,A) - E(-l)'' 2"^^/'^^^«p+r W^, 

where the sum is over r > and /i ~-+ A with — \X\ — r. 
Proof. The Giambelli formula for W(p,x) implies that 

j=2 a 

where C denotes the image of C(A) under the map which sends {i,j) to {i + + 
We obtain 

r>0 |Q|=r 

and then apply the mirror identity (|35p to finish the proof. □ 
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The reader should compare the stable Fieri rule (l34|) with the recursion formula 
P7p. This is illustrated in the next example. 

Example 6. For fc = 2 and A — (7, 4, 2, 1), we have 

Wj ■ W4,2,l = 1^7,4,24 + 2 1^8,3,2,1 + 2 1^8,4,2 + 2 1^9,2,24 + 2 ^9,3,1,1 + 4 ^9,3,2 

+ 2 14^10,2,1.1 + 4 VFlO,2,2 + 4 4^10,3,1 + 4 M^ll,2,l + 2 14^11,3 + 2 Wi2,2 , 

and hence 

W"7,4,2,l = ^7^^4,2,1 -Ws (2 1^3,2,1 + 2 W4,2) + W9 (2^2,2,1 + 2 M^3,i4 + 4 14^3,2) 

- wio (2 M^2,i,i + 4 1^2,2 + 4 T4^3,i) + (4 1^2,1 + 2 W3) - 2 W12 W2 • 

Setting fc = in Theorem [3] produces the following recursion formula for Schur 
Q-f unctions. 

Corollary 5. For any strict partition X and p > Ai, we have 

(38) Qip.M^) = 2"('/^)gp+.(x) Q^ix) 

where the sum is over r > and strict partitions fi G X such that X/ fi is a horizontal 
r-strip, andn(X/fi) equals the number of connected components ofX/ii. 

Observe that ([38|) is a generalization of Schur's identity 

Qa,bix) = Qaix) qb{x) - 2qa+i{x) qb-iix) + 2qa+2{x) qb-2{x) 

for any positive integers a, b with a > b. 

It would interesting to prove a top row recursion formula for the Wx polynomials 
analogous to ([57)1 in the general case, as well as for the V\ polynomials (in the Hall- 
Littlewood theory). Note that both of these recursions would have to interpolate 
between formula ([36)) . which arises when fc is sufficiently large for the W\ and when 
t = for the V\ , and formula ([38]) , which arises when fc = Oort = — 1, respectively. 

5. Reduction formulas and tableaux 

5.1. The mirror identity may be applied in the theory of Hall-Littlewood 
functions Q\{x ; t) of H2.3I to obtain a reduction formula; what is being reduced is 
the number of x variables used in the argument of Q\{x ; t). Let x = (x2, X3, . . .) 
and observe that qp{x ; t) = Y^^=o lii^i i Qp-i{^ ! t)- Therefore, for any partition 
A, we obtain 

(39) qx{x;t) = ^(Z„(2:i;t)(ZA-a(£;0 = ^ 4"' (1 - 9A-a(i ; 

summed over all compositions a. If R denotes any raising operator, we have 
Rq\{x ;t) ^ qRx{x ■,t) ^ ^ qa{xi ■,t) qjix^a{x ■,t) ^ ^ qa{xi ; t) qR^x^^) {i ; t) 

Q>0 a>0 

= ^ qa{xi \ t) Rqx^a{x]t). 
a>0 

Applying the Giambelli formula to (|39p. we thus deduce that 

oc 

(40) QA(:^;t) = 5]xl"l(l-t)#"QA-a(i;t) = 5]2:? ^ {I ^ t)*^Qx-a{x ;t) 

a>0 p=0 \a\=p 
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and hence, using ([29]) . the reduction formula 

oc 

(41) Qx{x;t)=Y.''i VA/M(OQM(i;0 

mI = |a|-p 

with the second sum over partitions /i C A such that A//i is a horizontal p-strip. 
Repeated application of the reduction equation (|41|) results in the tableau formula 
of [Ml 111.(5.11)] for the Hall-Littlewood functions. 

There is an alternative approach to the proof of (I4ip which applies standard 
symmetric function theory, as in e.g. [Ml 111.5]. If x' = {x'i,x'2, . . .) is a second set 
of commuting variables, then the equation 

Q\{x, ; t) = ^ Q\/f,{x ; t) Q^{x' ; t) 

summed over partitions /i C A may be used to define the skew Hall-Littlewood 
functions Q\/^ {x ;t). In particular, this gives 

Qx{xi,x;t) Qx/f,{xi ; t) Q^{x ; t). 

According to [Ml 111.(5.14)], we have 

(42) QA/M(a^i;0 = ^A/M(i)a;M 

and hence (|4ip is established. By comparing with (|40p. we can apply this reasoning 
to prove the mirror identity (|29)) for the Hall-Littlewood functions. Note however 
that the proof of (|42)) uses the inner product of [Ml HI. 4], and the point here is 
that this is not needed in order to establish Theorem [1] (which of course does not 
involve the variables x). The raising operator approach will be exploited further in 
the next subsections when we study theta polynomials. 

5.2. Let y = (yi, . . . , j/^.) and consider the theta polynomials "drix ; y) and Qx{x ; y) 
defined in ^3.3[ so that Q\ = -dx for any fc-strict partition A. We compute that 

oo ^ oc ^ k oc oc 

E ; - 1^ n 1^ 11(1 + y.^) - E 2*^ x\ Y: Mi ; y) 

r=0 ^ i=2 ' j=l i=0 r=0 

and therefore, for any /c-strict partition A, 

(43) i9A(a;;y) = ^xl"l2#"^A_„(£;y) 

summed over all compositions a. Applying the raising operator i?^ to both sides 
of (|43| produces 

oo 

e.(x;y) = ^4"'2#"e$L'](i;y) = ^xf ^ 2*" Q^i'lii ; y) , 

a>0 p=0 \a\=p 

where 6^^^^ — "dx-a by definition. We now use the mirror identity to 
deduce the next result. 

Theorem 4. For any k-strict partition \, we have the reduction formula 

oo 

(44) eA(x;y)=^x? ^ 2"(V^' e^(x ; j/). 
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Figure 3. Two fc'-relatcd boxes in a Young diagram 

5.3. In this subsection we will apply the reduction formula (|44p to obtain a tableau 
description of the theta polynomials 0a, where the tableaux in question are fillings 
of the Young diagram of A. We say that boxes [r, c] and [r',c'] are k! -related if 
|c — fc— ^|+r = |c' — A:— i|+r' (we think of fc' as being equal to — \). For 
example, the two grey boxes in the diagram of Figure [3] are fc'-related. We call box 
[r, c] a left box if c < k and a right box if c > k. 

If fi d X are two /c-strict partitions such that A//i is a fc-horizontal strip, we define 
Xq — fJ-o — oo and agree that the diagrams of A and fj, include all boxes [0, c] in row 
zero. We let R (respectively A) denote the set of right boxes of /i (including boxes 
in row zero) which are bottom boxes of A in their column and are (respectively are 
not) fc'-related to a left box of X/^i. 

Lemma 7. A pair fi d X of k-strict partitions forms a k-horizontal strip Xj \x if 
and only if (i) Xj \x is contained in the rim of X, and the right boxes of X/ fi form a 
horizontal strip; (ii) no two boxes in R are k' -related; and (Hi) if two boxes ofX/fi 
lie in the same column, then they are k' -related to two boxes of R, which both lie in 
the same row. The integer n{X/ fi) is equal to the number of connected components 
of A which do not have a box in column fc* + 1 . 

Proof. We have ^ ~^ A if and only if A ~> [p + r, fi) for any p > |A| + 2fc, where 
r = |A — Observe that a box of (p + r, ^) \ A corresponds to a box of fi which is a 
bottom box of A in its column. The fact that fi ^ X is characterized by conditions 
(i)-(iii) and that n{X/fi) = N{X, {p + r, /i)) is computed as claimed is now an easy 
translation of the definitions in H3.2I □ 

Let A and fi be any two fc-strict partitions with fi G X. 

Definition 2. a) A k-tableau T of shape X/fi is a sequence of fc-strict partitions 

^ = A° C A^ C • • • C A'' = A 

such that AVA*~^ is a fc-horizontal strip for 1 < i < r. We represent T by a filling 
of the boxes in X/ fi with positive integers which is weakly increasing along each 
row and down each column, such that for each i, the boxes in T with entry i form 
the skew diagram A'/A*^^. A standard k-tableau on X/fi is a fc-tableau T of shape 
X/ fi such that the entries 1, 2, . . . , |A — /i| each appear once in T. For any fc-tableau 
T we define 

n(T) = ^n(AVA*-i) and x^ = '[[x'P' 

i i 

where rrii denotes the number of times that i appears in T. 
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b) Let P denote the ordered alphabet {1' < 2' < ••• < k' < 1 < 2 < •••}. 
The symbols 1', . . . , fc' are said to be marked, while the rest are unmarked. A k- 
hitableau U of shape A is a filling of the boxes in A with elements of P which is 
weakly increasing along each row and down each column, such that (i) the marked 
entries are strictly increasing along rows and weakly increasing down columns, and 
(ii) the unmarked entries form a /c-tableau T. We define 

k 

n{U) = n{T) and {xyf ^ W y-' 

i=i 

where nj denotes the number of times that j' appears in U. 
Theorem 5. For any k-strict partition X, we have 

(45) e^{x;y) = Y,r^''Hxyf 

u 

where the sum is over all k-bitahleaux U of shape A 

Proof. Let to be a positive integer, a;^™^ = (xi, . . . , Xm), and let Qx{x'^'^^ ; y) be the 
result of substituting Xi = Q for i > to in Qx{x \y). It follows from equation ((44|) 
that 

oo 

(46) Qx{x^"^^-y)^Y.<^ E 2"(V^)e^(a;('"-i);2/). 

p— ^-^\ 

If I = |A|-P 

Iterating equation PS|) to times produces 

eA(a;('");y)=5]2"(^) x^e^(0;y) 

where the sum is over all fc-strict partitions /x C A and fc-tableau T of shape A//i with 
no entries greater than to, and 0^(0 ; y) is obtained from 8^ (a; ; y) by substituting 
Xi = Q for all i. The raising operator definition of gives 

(47) e^(0;y)=i?^e^(y) 

where = e^. (y) and er{y) denotes the r-th elementary symmetric polynomial 
in y. Since er{y) = for r > k, we deduce from (|T7)l that Q^{0 ;y) = unless /i is 
contained in the first k columns. But in this case we have C(/i) = and 

0p(O;y) = n(i-%)eM(2/) = v(y) 

i<j 

where /i' is the partition conjugate to fi. The combinatorial definition of Schur 
S'-functions (M* L(5.12)] states that 

(48) s,,{y)^J2y' 

s 

summed over all semistandard Young tableaux 5* of shape /x' with entries from 1 
to fc. In the spirit of this article, the identity may be derived using raising 
operators, by iterating the specialization of equation (|4T|) at t = 0. We conclude 
that 

eA(a;(");2/) = 5]2"(^) {xyf 
u 
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summed over all fc-bitableaux U of shape A with no entries greater than m. The 
result follows by letting m — > c». □ 

Example 7. Let fc = 1, A = (3, 1), and consider the alphabet Pi, 2 — {!' < 1 < 2}. 
There are twelve fc-bitableaux T of shape A with entries in Pi, 2- The bitableau 
1' 1 2 

T = ^ satisfies n{T) — 3, the seven bitableaux 

112 112 122 122 1'12 1'22 1'12 
12 12 2 1 1' 

satisfy n{T) — 2, while the four bitableaux 

I'll 1'2 2 I'll 1'2 2 
1 2 1' 1' 

satisfy n{T) = 1. We deduce from Theorem [5] that 

03,i(a;i,X2 ;yi) = {4x1x2 + Sxfxj + Axixl) + (2x1 + 8x^x2 + 8x1X2 + 22:^) yi 

+ {2x1 + AxiX2 + 2x1) yl 

= Q3,i{xi,X2) + {Qz{xi.,X2) + Q2,i(a;i,a;2)) yi + Q2(a;i,a;2) yl- 

5.4. Notice that (|45l) may be rewritten as 

e.(x;y)=^2"(^)x^V(y) 

with the sum over all partitions C A and fc-tableau T of shape A//i. This motivates 
the following definition. 

Definition 3. For A and ji any two fc-strict partitions with /i C A, let 

T 

where the sum is over all fc-tableaux T of shape A//i. 

Corollary 6. Let x — {xi^X2, ■ ■ ■) and x' — {x'i,x'2, ■ ■ ■) be two sets of variables, 
and let A be any k-strict partition. Then we have 

(49) e, (x, x' ; y) = ^ F^JJ^ (x) 9^ (x' ; y) , 

MCA 

(50) eA(x;y) = ^Fij;^(x)v(y), 
and 

(51) Ff (x,x') = 5]i^i;i(x)FW(x'), 
where the sums are over all k-strict partitions fj, d X. 

Proof. Let m be a positive integer, x*^™^ = (xi,...,Xm), and let -F^/^j (s:'-™-' ) (re- 
spectively 8a(x*^"') ;y)) be the result of substituting Xi — for i > m in F^f^j^ix) 
(respectively Qx{x ; y)). Applying equation (j46p as in the proof of Theorem [5] gives 

e.(x(™), x' ;y) = J2 Fx/M"^^) ^^i^' 5 v)- 
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Now let m ^ cxD to obtain (|49|) . Equations ([50]) and ([ST]) are deduced from (I49p by 
substituting x' = and y = 0, respectively. □ 

The polynomials 0^ (a;' ; y) for /i a fc-strict partition form a free Z-basis for the 
ring r''^^ (a;' ; y) of theta polynomials in x' and y. The identity (|49)) states that 
-^A/)t(^) coefficient of Qi_i{x' ; y) when Qx{x,x' ;y) is expanded in this basis. 

We deduce that F['j^^{x) is a symmetric function in the variables a;. The identity 
(ICTI) may be further generalized as follows. 

Corollary 7. For any two k-strict partitions A and fj, with fj, <Z X, we have 

(52) 4%M-T.Pi/l(-)Pl/>')' 

where the sum is over all k-strict partitions v with ^ (Z v <Z X. 
Proof. Let z = (zi, Z2, . . .) be a third set of variables. We have 

We now equate the coefficients of 8^(2; ; y) at either end of the above equalities. □ 

In the next section, we will show that the fc-strict partitions v which contribute 
positively to the sum in equation ()52p correspond to the reduced factorizations of 
a certain skew element in the hyperoctahedral group. 

Example 8. a) Suppose that fc = or, more generally, that /i^ > min(fc, Xi) for all 
i. Then Dcfinition[3]becomes the tableau based definition of skew Schur Q-functions 
m 111.(8.16)], hence F^'j^ix) - Qxf^{x). 

b) Suppose that C(A) — 0, so in particular Xi > k implies that i — 1. One then 
checks that n{X/ ji) is equal to the number of edge-connected components of X/ ^. 
For any partition jJ. d X, Worley |W1 §2.7] has shown that 

^2«(^).^ = 5v,(x), 

T 

where the symmetric function S\/^(x) satisfies 

S\/t,{x) = det(gA.-p,+j-»(a;))jj. 

We therefore have F^^^{x) — Sx/^{x) and 

0A(a; ; y) = X! '5'a/m(^) V (2/)' 

mca 

in agreement with jBKT2| §5.5]. 

c) It is clear from Definition [3] that in general, the skew function -f^/^Ca^) depends 
on both A and /i, and not only on the difference A \ /i. For instance, we have 
-^(3^ 2)/(3) (^) ~ while F(j\)/(^r)^x) ~ Q'ii^) foi' ^iiy > 3. On the other hand. 
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■^(5,4,1, i)/(4,3)(^) = -^(A,5,4,i,i)/(A,4,3)(2^) = ^o^' strict partition A with \i > 5. 
Criteria for the vanishing of ^"^/^^(a;) are given in Proposition [H] and Corollary [TUl 

d) Suppose that there is only one variable x. Then we have 

^(fc) _ f2"(-^/'') x'-^-'^l if A//X is a ^-horizontal strip, 
x/t,W - Otherwise. 

The reader should compare this to (|42p. 

5.5. Observe that we have 

p 

dp{x, x'; y) = ^ qi{x) dp-i{x';y) 

i=0 

and therefore, for any fc-strict partition A, 

■dx{x, x'; y) = ^ qa{x) i}x-a{x'; y) 

Q>0 

summed over all compositions a > 0. It follows that 

(53) Qx{x,x']y) = ^ 9^(2;) e^L^^(x'; y). 

For each fixed composition a, the product qa{x) — W^qaiix) is a nonnegative 
linear combination of Schur Q-functions. In addition, Q^_^{x ]y) is a Z-linear 
combination of the Qfj,{x' ;y) for fc-strict partitions /i. By comparing ()53|) with 
(|49l) . we deduce that for every two fc-strict partitions A, /i with /i C A, the function 
i^l^^(x) is a linear combination of Schur Q-functions with integer coefficients. In 
the next section, we will prove that these coefficients are nonnegative integers. 

6. Stanley symmetric functions and skew elements 

6.1. Let Bn be the hyperoctahedral group of signed permutations on the set 
{l,...,n}, and Boo — U„-B„. We will adopt the notation where a bar is writ- 
ten over an entry with a negative sign. The group Boo is generated by the simple 
transpositions Si — (i, i + for i > 0, and the sign change so(l) — 1. Every element 
w £ Boo can be expressed as a product of simple reffections Si ; any such expression 
of minimal length is called a reduced word for w. The length of w, denoted i{'w), is 
the length of any reduced word for w. A factorization w — uv in Boa is reduced if 

£{w) =£{u)+£(v). 

Following |FSj and [FK1[ IFK2| . we will use the nilCoxeter algebra Bn of the 
hyperoctahedral group i?„ to define type C Stanley symmetric functions and Schu- 
bert polynomials. Bn is the free associative algebra with unity generated by the 
elements ug, ui, . . . , modulo the relations 

m| = i>0 
UiUj — UjUi \i — j\ > 2 

UiUi+iUi = Ui+iUiUi+i i > 

Ui^UiUqUi — UiUqUiUq. 

For any w G Bn, choose a reduced word Sai ■ ■ ■ for w and define = . . . Ua^- 
Since the last three relations listed are the Coxeter relations for Bm it is clear that 
Uw is well defined, and that the for w € Bn form a free Z-basis of S„. 
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Let u) be an indeterminate and define 

Ai{u:) = (1 + LOUn-\){\ + lOUn-2) ' ' ' (1 + •^Ui) ; 
C(cj) = (1 + U)Un~\) • • • (1 + a;Ui)(l + 2ll1Uq)(\ + CJUi) • • • (1 + IjJM„_i). 

Set X = (xi, a;2, . . .) and consider the formal product C{x) := C{xi)C{x2) ■ ■ ■ ■ 
Arguing as in [FK2, Prop. 4.2], we see that the relation C{xi)C{xj) = C{xj)C{xi) 
holds for all indices i and j. We deduce that the functions Fw{x) in the formal 
power series expansion 

(54) C{x) = J2 Fn,{x)u^ 

are symmetric functions in x. The F^, are the type C Stanley symmetric functions, 
introduced and studied in [BH } [FK2l IL] . 

Let y = {yi,y2, ■ ■■)■ The Billey-Haiman type C Schubert polynomials €io{x;y) 
for w G Bn are defined by expanding the formal product 

(55) C{x)Ai{yi)A2{y2) ■ ■ ■ An-iivn-i) ^ ^ €w{x\y)u^,. 

The above definition is equivalent to the one in [BHj . as is shown in }FK2| §7]. One 
checks that C^, is stable under the natural inclusion of i?„ in Bn+i, and hence well 
defined for w € Boo- We also deduce the following result from and ([55]) . 

Proposition 3. Let w G Boo o,nd x' ~ (x'i,x'2, ■ ■ ■)■ Then we have 

(56) F^{x,x')^ Pu{x)F,{x') 

UV — W 

and 
(57) 

;y)^ Fu{x)U{.x' -y) 

UV — W 

where the sums are over all reduced factorizations uv = w in Boo ■ 

6.2. We say that an element w £ Boo is k-Grassmannian if it is Grassmannian 
with respect to the simple reflection Sk, i.e., if £{wsi) — £{w) + 1 for all i ^ k. 
Let V{k,n) denote the set of fc-strict partitions whose Young diagrams fit inside 
an (n — k) X (n + k) rectangle. Given any fc-strict partition A, choose n such that 
A G V{k,n). We associate to the pair (A, n) a fc-Grassmannian element wx G B„, 
according to the following prescription. 

The signed permutation w\ — {wi, . . . , w„) has a unique descent at fc, that is, 
w{i) < w{i + 1) whenever i ^ k. For A e V{k, n) we let A^ be the strict partition 
formed by the boxes of A in columns fc + 1 through k -\- n. The negative entries 
of w\ are then given by the parts of A^. Consider the shape n(fc,n) obtained by 
attaching an (n — fc) x fc rectangle to the left side of a staircase partition with n 
rows. When n = 7 and fc = 3, this looks as follows. 



n(fc,n) = 
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The diagram of A can be placed inside Ii{k, n) so that the northwest corners of 
A and n(fc, n) coincide. The boxes of the staircase partition which are outside A 
then faU into south-west to north-east diagonals. The first k (respectively, the last 
n — k — ^(A^)) entries of w\ are the lengths of the diagonals which are (respectively, 
are not) fc-related to one of the bottom boxes in the first k columns of A. For 
example, the partition A — (8, 5, 2, 1) G 7^(3, 7) results in the element w\ — 1475236. 



A = 
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Conversely, for any /c-Grassmannian element w G i?„ there exists strict partitions 
u, C, V of lengths fc, ^, and n — k — £, respectively, so that 

w = {Uk, ■ ■ • ,ui,Ci, ■ • ■,Ci,Vn-k-e, ■ ■ ■,Vl). 
Define ai for 1 < « < fc by 

ai^Ui+i-k-l + #{j I Cj > Ui}. 

Then w = w\ for the partition A G Vik, n) with A^ = C and such that the lengths 
of the first k columns of A are given by ai, . . . , Offc. 
It was proved in [BKT2i §6] that 

(58) ex{x;y)^€^xix;y) 
for any fc-strict partition A. 

6.3. The following definition can be formulated for any Coxeter group, but the 
name is justified by its application in the case of the finite classical Weyl groups. 

Definition 4. An element w G Sqo is called skew if there exists a fc-strict partition 
A (for some fc) and a reduced factorization w\ — ww' in Boo- 

Note that if we have a reduced factorization w\ — ww' in i?„ for some partition 
A G V{k,n), then the right factor w' is fc-Grassmannian, and therefore equal to 
for some partition /x G V{k,n). 

Proposition 4. Suppose that w is a skew element of Boo, o,nd let A and fi be k- 

strict partitions such that the factorization wx = wWf^ is reduced. Then we have 

/i C A and Fy^{x) — F^'j^^{x). 

Proof. By combining ([58]) with ([49]) and ([ST]) . we see that 

(59) ^F^5;,(x)e^(x';y) = eA(x,x';2/)= ^ Fu{x) (^^x' ; y) 

fiCX uv=wx 

where the second sum is over all reduced factorizations uv — w\. In any such 
factorization, the right factor v is equal to Wu for some fc-strict partition i/, and 
therefore €y{x' ; y) = Qu(x' ; y). Since the 8^(2;' ; y) for v a fc-strict partition form 
a Z-basis for the ring F^*^) {x' ; y) of theta polynomials in x' and y, the desired result 
follows immediately. □ 
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The Fieri rule (j22|) illustrates that the order relation on /c-strict partitions given 
by the inclusion of Young diagrams is not compatible with the Bruhat order on Boo ■ 
However, Proposition!?] shows that the weak Bruhat order on the fc-Grassmannian 
elements of Boo respects the inclusion of fc-strict diagrams. 

Definition 5. Let A and /i be fc-strict partitions in V{k,n) with fi C X. We say 
that (A, /i) is a compatible pair if there is a reduced word for wx whose last \fi\ 
entries form a reduced word for w^; equivalently, if we have £{w\'w~^) — |A — In 
other words, (A, fj,) is a compatible pair if wx exceeds lu^t in the weak Bruhat order. 

Corollary 8. Let (A,/i) be a compatible pair. Then there is a 1-1 correspondence 
between reduced factorizations of w\w'^^ and k-strict partitions v with fi G v <Z X 
such that (A, v) and (v, /i) are compatible pairs. 

Proof. Suppose that w\ — wWf^ and let w — w'w" be a reduced factorization. Then 
w"w^ is fc-Grassmannian and hence equal to w^, for a unique fc-strict partition v. 
Proposition 3] implies that fi C i' d X, and clearly the pairs (A, v) and {v, fi) are 
compatible. The converse is obvious. □ 

Theorem 6. Let X and fi be k-strict partitions in V{k,n) with fi C X. Then the 
following conditions are equivalent: (a) 7^ 0; (b) (A, /i) is a compatible pair; 

(c) there exists a standard k-tableau on Xj \i. If any of these conditions holds, then 

Proof. Equation ()59p may be rewritten in the form 

(60) E ^vl(-) 0m(^' ; = E (-) 0m(^' ; y) 

fJ.C\ /J 

where the second sum is over all /i C A such that (A, /i) is a compatible pair. It 
follows that 



,(fc) _ J K^w-^ (^) if i^^ is a compatible pair, 
otherwise 



^i/i(-) 



and hence that (a) and (b) are equivalent. Suppose now that (A, /x) is a compatible 
pair with |A| = + 1, so that w\ — SiW^ for some i > Q. Observe that if x is 

a single variable, then F^.^x) — 2x, and therefore F'^j'^{x) ^ 2x ^ Q. We deduce 
from Example md) that A//i must be a fc-horizontal strip. Using Corollary [H it 
follows that there is a 1-1 correspondence between reduced words for wxw~^ and 
sequences of fc-strict partitions 

M = A° c A^ C • • • C A'^ = A 

such that |A*| = |A'~^|-|-1 and AVA*^^ is a fc-horizontal strip for 1 < i < r = |A — 
The latter objects are exactly the standard fc-tableaux on X/fj,. This shows that 
(b) implies (c), and the converse is also clear. □ 



The previous results show that the non-zero terms in equations (|49| and ((5T|) 
correspond exactly to the terms in equations (j57p and (j56[) . respectively, when 
w = w\ is a Grassmannian element of Boo- 

Corollary 9. Let w G Boo be a skew element and (A, fi) be a compatible pair such 
that w\ = ww^. Then the number of reduced words for w is equal to the number 
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of standard k-tableaux on A//i and to the coefficient of X1X2 • ■ ■ Xr in 2 ^P\/^{x), 
where r = |A — /i| = £{w). 

Example 9. Let A be a fc-strict partition, A^ be defined as in H6.21 A^ = A \ A^, 
and /i C A^. We can form a standard /c-tableau on X/ ^ by filling the boxes of A'^///, 
going down the columns from left to right, and then filling the boxes of A^, going 
across the rows from top to bottom. If /i C A but fii > k, this procedure does 
not always work, see for instance Example [HJc) . When A; = 3, A = (8,6,5,2), and 
H = %, the 3-tableau on A which results is 

1 5 9 12 13 14 15 16 

2 6 10 17 18 19 

3 7 11 20 21 

4 8 

which corresponds to the reduced word 

Si So S2 Si So S4 S3 S2 Si So S3 S2 Si S5 S4 S3 S2 Sg S5 S4 S3 

for the Grassmannian element wx = 1675324 G -87. 

A sequence (ii, . . . , i„i) is called unimodal if for some r with < r < m, we have 
ii > i2 > ■ ■ ■ > ir < ir+i < ■ ■ ■ < im- An element w 6 i?oo is unimodal if it has a 
reduced word s^^ • • • Si^ such that {ii, . . . ,i„i) is a unimodal sequence. A tableau 
T has content given by the composition a if ai of the entries of T are equal to i, 
for each z > 1. We can now state the following generalization of Corollary [9l 

Proposition 5. Let w G -Boo be a skew element and (A, /i) he a compatible pair such 
that w\ = wWfj^. Then there is a 1-1 correspondence between reduced factorizations 
Ui---Ur of w into unimodal elements Ui and k-tableaux T of shape A//i with r 
distinct entries. The lengths of the Ui agree with the content of T , and the number of 
reduced words for w obtained by concatenating unimodal reduced words for ui, . . . ,Ur 
of the corresponding lengths is equal to 2"(^)~'". 

Proof. From the definition of i^^, in i i6.ll it follows that if there is only one variable 
X and w ^ 1, then we have Fw{x) — 2nwX^^'^'\ where denotes the number of 
unimodal reduced words for w. Moreover, for each to > 1 we have 

(61) 

III ■ ■■Ujn—W 

summed over all reduced factorizations ui ■ ■ ■ Um for w. The result follows by com- 
paring ([6T|) with Definition [3] and using Corollary [H □ 

Let A G V{k, n) and w\ G i?„ be the corresponding fc-Grassmannian element. There 
are analogues of Corollary [9] and Proposition [5] for Qx{x;y) and the A:-bitableaux 
of shape A. We say that a permutation w G iSn is decreasing if v has a reduced word 
Sii ■ ■ ■ such that zi > • • • > im. Then the /c-bitableaux of shape A correspond to 
reduced factorizations ui ■ ■ ■ UrVi ■ ■ ■ Vg of w\ with the Ui G i?„ unimodal and the 
Vj G Sn decreasing. We leave the details to the reader. 

According to |BH] and [L] , the type C Stanley symmetric function F„ is a non- 
negative integer linear combination of Schur Q-functions. Together with Theorem 
[6l this implies the following result. 

Corollary 10. For any two k-strict partitions X, 11 with C A, the function 
P\f (x) is a nonnegative integer linear combination of Schur Q-functions. 
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6.4. We say that a permutation w is jully commutative if any reduced word for vj 
can be obtained from any other by a sequence of braid relations that only involve 
commuting generators. It follows from [Stei Thm. 4.2] that a permutation w is 
fully commutative if and only if there exists a Grassmannian permutation wx and 
a reduced factorization w\ — tuw' for some permutation tu' . In other words, the 
skew elements of the symmetric group are exactly the fully commutative elements. 
Following [FS| . the expansion of the formal product 

Ai{xi)Ai{x2) ■ ■ ■ ^ ^ G^{x)u^ 

may be used to define the type A Stanley symmetric functions G^{x) for G S'„. 
Stanley [Staj introduced to study the set of reduced words for w. It is shown in 
[BJSl §2] that if 137 G Sn is fully commutative (or equivalently, 321-avoiding) then 
Gzo ~ *A//i is a skew Schur function, and the number of reduced words for -w is 
equal to the number of standard tableaux on A//i. Our present definition and study 
of skew elements in the hyperoctahedral group is therefore completely analogous to 
this established theory for the symmetric group. 

Any fully commutative element of i3„, in the sense of |Ste| . is a skew element. 
However the converse is emphatically false, for example the 1-Grassmannian ele- 
ment W(4.i) = 231 is not fully commutative. The three reduced words 

S1S2S1SQS1 S2S1S2SQS1 S2S1SQS2S1 

for 231 correspond respectively to the three standard 1-tableaux 

1234 1245 1345 

5 3 2 

on the diagram A — (4, 1). 
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